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I. ABSTRACT 

This paper describes the recent developments of techniques which assist an operator in the control of 
remote robotic systems. In particular, applications are aimed at two specific scenarios: (1) the control of remote 
robot manipulators; and (2) motion planning for remote transporter vehicles. Common to both applications is the 
use of realistic computer graphics images which provide the operator with pertinent information. This paper will 
describe the specific system developments for several recently completed and ongoing telepresence research projects 
in the Center for Intelligent Machines and Robotics (CIMAR) at the University of Florida. 


II. INTRODUCTION 

Significant advances have been made in a broad spectrum of component technologies such as kinematics 
and dynamics, control, vision and pattern recognition, obstacle avoidance, computer graphics, and artificial 
intelligence. Each of these technologies can individually impact on an operator’s efficiency in the control of a 
remote manipulator or transporter. An objective of telepresence systems is to combine these component 
technologies in order to provide the operator with multiple sensory feedback data. In this manner an operator 
can in effect directly experience the environment of the remote manipulator system. The abundance of sensory 
feedback coupled with computer assisted operation (a low level of autonomy) is what distinguishes telepresence 
from earlier teleoperated systems. 

The operational scenario of remote systems may vary significantly from case to case. In some situations, 
the operator may be required to control the remote system in real time for which it is necessary for the remote 
system to respond directly to operator input. In many circumstances, however, a real time response is not possible 
or desired. A significant communication time delay, for example, can make real time control awkward at best 1 ' 1 . 
A second classification can be made regarding the operational environment. Clearly, it is necessary to know whether 
or not the operational environment of the remote system is known a priori. Knowledge of an accurate model of 
the environment can significantly impact the type of telepresence system which is to be implemented. Operation 
in a structured environment such as a nuclear power plant ,2 ' is considerably different from controlling a remote 
transporter in the field. 


‘Numbers in brackets refer to references at end of paper. 
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This paper will describe several projects conducted at CIMAR which all deal with the control of remote 
manipulator or transporter systems. For each case, a scenario is established where: (1) the operator is either 
directly controlling the system in real time or is performing an off-line planning task; and (2) the environmental 
descriptive model is known a priori or the system is operating in a totally unknown area. 

111 CONTROL OF REMOTE ROBOT MANIPULATORS 
A. Initial Telepresence System ,3) 

The first telepresence system developed at CIMAR was concerned with the real time control of a remote 
manipulator in an unstructured environment. It was assumed that the operator could not directly see the 
manipulator or the objects in its workspace as he was directly maneuvering the robot with a joystick device. The 

task for the operator was to acquire and move several cubes in the workspace without coming into contact with a 
sphere. 


The primary sensory feedback required to perform this task is vision. In many instances, vision alone would 
give the operator sufficient information in order to manipulate the cubes without contacting the sphere. Logically, 
this vision could be provided by video cameras which surround the workspace of the manipulator or which are 
attached to the manipulator itself. It should be noted, however, that the use of video cameras to provide vision 
feedback does have certain disadvantages. Primarily, more than one camera must be used to provide sufficient 
viewing positions. Each of these cameras will most likely have at least three degrees of freedom (zoom, tilt, and 
pan) thereby giving the operator numerous additional parameters to control. Furthermore, environmental conditions 
may cause the video image to be blurred or poor lighting and contrast may make the image confusing and unclear. 

Because of these limitations, the goal of this project was to evaluate the operator’s performance when the 
vision feedback was provided via a computer graphics display. The use of computer graphics offers three distinct 
advantages. First, images are clear and sharp since colors and contrast can be selected and programmed. Second, 
the computer graphics system allows for the viewing of the image from any desired vantage point. This ability 
removes the requirement for a multitude of video cameras and allows the operator to focus attention on only one 
monitor screen. Third, the computer graphics system can provide additional feedback to the operator concerning 
imminent collisions. When the manipulator was moved close to the spherical obstacle, the color of the sphere was 
changed and an audio tone was emitted. This additional feedback regarding collisions significantly improved 
operator confidence and performance. The collision warning feature was implemented by placing the obstacle in 
an imaginary rectangular protective box. The graphics system in effect compares the image of the robot with this 
protective box and determines in real time whether any part of the robot is in contact. 


A photograph of the initial telepresence system is shown in Figure 1 while the system configuration is shown 
in Figure 2. The system consisted of the following four component technologies: 

(1) Robot manipulator. 

(2) Six degree of freedom universal joystick. 

(3) Obstacle detection vision system. 

(4) Computer graphics system. 


The manipulator was drawn on the graphics screen by communicating the joint angles from the manipulator to 
the graphics workstation at every instant. The vision system was used to locate the objects in the manipulator 
workspace so that they could be displayed on the graphics screen. Shown in Figure 3 is an image of the robot as 
it comes close to the obstacle. 
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Figure 3: Warning of Imminent Collision 


The initial telepresence system showed that it was feasible and practical to utilize a computer graphics 
display to control a remote robot manipulator in real time in an unknown environment. The clarity of the image 
coupled with the real time determination of imminent collisions with obstacles significantly improved operator 
performance. 


B. Robot Path Processor 

As an extension to the previous project, an investigation of off-line task planning was conducted. For this 
project it was assumed that the location of objects in the workspace of the robot were known (either from a priori 
knowledge of a world model or from a sensor scan of the workspace). Off-line task planning offers an advantage 
if communication delays prohibit real time operation, or if an operator can plan a task much quicker than the 
manipulator can execute it. Further, off-line planning offers the operator the option to edit or modify the task 
before it is sent to the manipulator for execution. 

During this project, emphasis was placed on the development of the task editor. The objective was to be 
able to allow the operator to rapidly enter a robot task and then be able to refine it. The resulting system consists 
of the same universal joystick controller and computer graphics workstation used in the previous project together 
with a PUMA 560 industrial robot. The operator moves the animated robot on the screen and then can go back 
and perform such tasks as moving specific points on the path or modifying certain sections to be perfectly straight. 

One problem that had to be overcome, however, was how to move the joystick in order to get a desired 
motion of the robot on the graphics screen. Confusion resulted in that the operator could look at the robot from 
any viewing position and then had to mentally reference the joystick and manipulator coordinate systems (see 
Figure 4) . This problem was overcome by introducing an additional coordinate transformation which mapped the 
joystick coordinates into the screen coordinates. The result was that a movement of the joystick to the right or left 
would cause the end effector of the robot to also move to the right or left of the graphics screen, and so on. In this 
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Figure 4: Coordinate System Reference Frames 


manner, the operator could always correlate the motion of the robot in response to any joystick move, no matter 
how the scene was being viewed. This feature significantly improved the off-line planning process and was also later 
applied to the real time control scenario. 


C. Off-Line Programming with Autonomy |S| 

Off-line task planning can be significantly enhanced by the introduction of autonomy. In this project it was 
again assumed that the location of objects in the workspace of the robot were known, either from a priori knowledge 
of a world model or from a sensor scan of the workspace. The objective of this effort was to develop algorithms 
which would autonomously generate robot motions and tasks based on high level user inputs. 


The development of motion planning algorithms which avoid obstacles was a major focus of this effort. 
Shown in Figure 5 is a path which has been generated in order to move a manipulator between two user specified 
positions. A significant feature of the algorithm is that resulting paths are generated rapidly (under five seconds 
of computation time) (6) . The algorithms take advantage of the knowledge of the geometry of the manipulator and 
its environment in order to avoid excessive heuristic searches. 


Common to all off-line approaches, the operator is given the opportunity to preview the planned task on the 
graphics screen prior to the manipulator beginning the task. In this manner, the operator can evaluate and critique 
the performance of the autonomous algorithms. 

This research program was conducted for the U.S. Army Belvoir R,D,&E Center, Fort Belvoir, Virginia. 
The specific application for this project involved logistics operations and is shown in Figure 6. A camera on board 
the fork attachment was able to locate a target mounted on a pallet and then calculate the position and orientation 
of the pallet relative to the camera. With this information, the operator would be able to acquire the pallet and 
maneuver it off-line via the graphics animation system. Once a task was planned, previewed, and approved, the 
manipulator would be commanded to perform the operation. 
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Figure 5: Autonomously Generated Path 
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Figure 6: Logistics Application 
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IV. MOTION PLANNING FOR REMOTE TRANSPORTER VEHICLES 

Many of the man/machine interface control issues associated with the operation of remote transporter 
systems are directly analogous to those already discussed for manipulator systems. In particular, the transporter 
operation can be categorized according to whether or not it moves in real time under operator control and as to 
whether or not a prior knowledge of the environment is known. 

The work in this area is sponsored by the U.S. Department of Energy. The University of Florida, in 
cooperation with the Oak Ridge National Laboratory and the Universities of Michigan, Tennessee, and Texas is 
developing an advanced robotic system under the University Program for Robotics for Advanced Reactors. As such, 
the developments in this area are specifically aimed at the operation of a transporter vehicle in a nuclear power plant 
environment where it is assumed that an accurate world descriptive model exists. 

A. Development of an Articulated Transporter/Manipulator System (AT MS) t?1 

The complex obstacle strewn environment of a nuclear power plant requires that the robotic system be 
capable of crossing or jumping over substantially sized obstacles. Transport mechanisms consisting of combinations 
of wheels, tracks, and legs were considered as candidates, but the inherent mobility limitations of these mechanisms 
led to the selection of a transporter comprised of multiple articulated segments (see Figure 7). 

The ATMS is comprised of eighteen individual segments which provide both maneuverability and locomotion. 
Each segment is twenty four inches in length, and has a pair of motor driven wheels to provide traction for forward 
and reverse motion. Segments are connected in series by a pair of revolute joint axes. The significant feature of 
the design is that the ATMS will be able to cross over horizontal gaps of up to twelve feet in length. 



Figure 7: ATMS Model 
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Figure 8: Animated Representation of ATMS 


Initial control of the ATMS has been established by having an operator specify the direction and velocity 
of the lead segment. Subsequent segments of the ATMS will follow the path of the segment directly in front of 
it (see Figure 8). Work to date includes the development of follow-the-leader algorithms for vertical and horizontal 
navigation. Further, a series of stability rules have been generated which govern the effective maneuverability of 
the system. During a man-controlled real time operation of the system, the stability rules will be continuously 
evaluated and monitored, and the operator will be warned of potentially unstable actions. This combination of direct 
man-controlled operation with computer assistance is a fundamental principle of the telepresence concept. In 
addition to the real time human control, off-line autonomous control algorithms are currently being implemented. 

B. HERMIES Implementation 

A fully operational version of the ATMS has yet to be fabricated. In order to demonstrate the progress of 
the research program, the HERMIES robotic transporter at the Oak Ridge National Laboratory is being used as 
a system demonstrator (see Figure 9). HERMIES is limited to performing planar motion. However, many of the 
man-machine control issues which will involve the ATMS can be resolved via HERMIES. 

The system demonstration combines off-line and real time operation. During the planning phase, the 
operator can plan a motion path via (1) the computer graphics system alone (Univ. of Florida), (2) a three degree 
of freedom force reflecting joystick (Univ. of Texas), or (3) an autonomous path planning algorithm (Univ. of 
Michigan). Once a path is planned, the operator is able to review it and modify it if necessary prior to HERMIES 
beginning its motion. Figure 10 shows the animated display provided to the user when a path is being planned. 
Three interactive windows are provided which present a top view, the view from HERMIES, and the view from a 
user controlled camera located anywhere in the environment. 

An important aspect of the planning phase is the time acceleration concept^. The operator can effectively 
specify a time scale factor so that the planning phase can be accomplished in an accelerated time scale. For 
example, the ATMS can achieve a maximum velocity of only two feet per second. By selecting an accelerated time 
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Figure 10: Operator’s Console Image 
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scale, the operator will be able to perform the motion planning as if the ATMS had an unlimited maximum velocity. 
Upon the actual motion of the ATMS, the plan would be reconstructed in the real time domain. 

V. CONCLUSION 

Several recent and ongoing research programs have been described which all deal with some aspect of 
telepresence system development. Varying techniques have been discussed which assist an operator in the control 
of remote robotic systems either in real time or off-line, or when a prior knowledge of the environment is known 
or not. 


It appears evident that any resulting telepresence system must be able to perform in ail of the operational 
categories. For example, although a nuclear power plant may be a structured environment for navigation purposes, 
the environment can become unstructured during a manipulation task. Because of this, the generic techniques that 
were developed here can be combined together in order to offer the operator a highly sophisticated man-machinc 
interface. 
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